Overview

Motivation: Recall our motivation originally stemming from an interest in the “relative” volatility of fossil fuel versus renewable prices. After repeated discussion regarding the comparability of the two types of energy sources and scoping of the data available we pivoted to questions more directly related to “explaining”/investigating the recent spike in electricity prices (both in level and volatility) following a surge in natural gas prices and the Russian invasion of Ukraine and how it might have been mediated (or not) by increased renewable energy generation.

RQs: Though still to be workshopped, potential guiding questions along these lines (most follow from our past conversations):
1. To what extent, if any, does increased reliance on renewable sources of electricity insulate consumers or industry from the effects of spikes in electricity price inputs (ie. natural gas spike post-Ukraine)?
2. What are the costs (to electricity consumers - industrial and/or household) of recent increased price volatility and to what extent is greater renewable energy as a share of generating capacity a mediating or aggravating force?
3. Setting the price is an economic question - how efficiently does the market translate the true cost into price signals (ie. shifting share to lower cost renewables not yet filtering into price signals due to marginal price-setting in the market)?
4. What is the potential efficiency gain of switching to renewables even if it is not yet translated into prices?
5. Exogenous: use the connection of large wind or solar connections to the grid as plausible exogenous variation. See if there have been changes in capacity and whether they had any detectable effect on prices, all else equal.

Data: The below documents the data currently pulled to attempt a preliminary mean model of hourly wholesale electricity prices in Germany: * Wholesale Electricity Prices (Hourly - EMBER)
* Electricity Demand/Load (Hourly - ENTSOE)
* Electricity Generated by Energy Source (Hourly - ENTSOE)
* Various Commodity Price Indices (Monthly - IMF)
* Total and Country-Specific Net imports (Hourly - ENTSOE)
* Day-ahead Solar and Wind Forecasts (Hourly - ENTSOE)
* Redispatching Costs (“Hourly” - netztransparenz.de - they are reported at a particular hour but are contracted for “implementation” over a certain multi-hour period, I think…as mentioned, to be investigate more)

Data that remains to be integrated:
* Auction Bid Data (Data on electricity auction bids - this would likely 1. help us determine if renewables set the market price (if ever) and 2. the variability in generation costs of fossil fuel providers)
* Economic Activity (GDP or VAT)

Data Issues: Several data issues cropped up in the process. I note the most glaring issues in bold throughout this notebook. Notably, the main issues are:
* the lower-frequency of certain candidate explanatory variables (ie. fuel and non-fuel commodity prices and redispatching costs)
* missing data on net imports from France and Denmark in specific years (data exists but is not complete)
* how to aggregate the data on redispatching costs which are reported with hourly specificity but with different summarising metrics and “implementation horizons” (ie a load-balancing request made with a 4-hour implementation horizon). Fortunately, it does nto seem to be an issue of data availabilty but rather deeper thinking about what measure (and harmonisation across energy types) is most relevant for our ultimtate research question.

Model Attempt: Preliminary model attempt bringing this data together using gets under the “approaching a model…” tab.

Additional thoughts: Though, ultimately, it would be very ineresting to broaden to a multi-country scope (especially if this might allow for some comparative analysis on variable country-specific shares of renewable electricity generation), we decided to focus on Germany as a first case to see how we go. If/when we take the step to further countries, any data that I pull from ENTSO-E has an accompanying scraping script that could (reasonably) easily be repurposed to pull the identical data on other countries.

Start simple: Germany Only

Data

Using data from EMBER on European wholesale electricity prices (hourly - daily and monthly also exist). The hourly prices are the day-ahead wholesale prices in Germany.

Electricity Demand/Load

Total load per bidding zone per market time unit - in our case, total load, per country, per hour in MW.

Source

The following is data from ENTSO-E on hourly electricity loads.

Components: Actual total load (including losses without stored energy) = net generation – exports + imports – absorbed energy

Generation by Resource

Actual Generation per Production Type from ENTSOE This value represents the actual aggregated net generation output in MW per market time unit (hour) and per production type.

Note: There is 1 observation missing per year (either 0 or NA) on a late day in march at 2 am….a cleaning issue? Something silly?

Fuel and Commodity Prices

IMF provides monthly commodity prices and price indices.
Source: https://www.imf.org/en/Research/commodity-prices

The commodity prices listed below are:

Note: These are MONTHLY values. I fill these values to hourly when merging the data later on.

I include the following three commodity prices in the model selection for now (final plot in this section):
* Natural Gas, Netherlands TTF Natural Gas Forward Day Ahead, US$ per Million Metric British Thermal Unit
* Fuel & Non-Fuel Commodities: All Commodity Price Index, 2016 = 100, includes both Fuel and Non-Fuel Price Indices
* Indust. Input Price: Industrial Inputs Price Index, 2016 = 100, includes Agricultural Raw Materials and Base Metals Price Indices

Note: We should probably rethink the non-natural gas prices (the latter two indices). The two latter indices are global indices and given the industrial structure of Germany we might want to consider a Producer Price Index for Germany specifically, assuming that exists? Additionally, as you can see in the graph, the latter two essentially trace each other/collinearity issues.

Commodity Type Index Label (in plots) Index Definition Index Unit (USD or Index)
Coal Coal Coal Price Index, 2016 = 100, includes Australian and South African Coal Index
Gas Nat. Gas (NDL TTF) Natural Gas, Netherlands TTF Natural Gas Forward Day Ahead, US$ per Million Metric British Thermal Unit USD
Oil Oil - Avg Brent, West TX, Dubai (USD) Crude Oil (petroleum), Price index, 2016 = 100, simple average of three spot prices; Dated Brent, West Texas Intermediate, and the Dubai Fateh USD
Oil Oil - Brent Crude Oil (petroleum), Dated Brent, light blend 38 API, fob U.K., US$ per barrel USD
Oil Oil - Dubai Crude Oil (petroleum), Dubai Fateh Fateh 32 API, US$ per barrel USD
Oil Oil - West TX Crude Oil (petroleum), West Texas Intermediate 40 API, Midland Texas, US$ per barrel USD
Oil Oil - Avg Brent, West TX, Dubai Crude Oil (petroleum), Price index, 2016 = 100, simple average of three spot prices; Dated Brent, West Texas Intermediate, and the Dubai Fateh Index
Gas Nat. Gas Henry Hub Natural Gas, Natural Gas spot price at the Henry Hub terminal in Louisiana, US$ per Million Metric British Thermal Unit USD
Composite Commodity Indices Indust. Input Price Industrial Inputs Price Index, 2016 = 100, includes Agricultural Raw Materials and Base Metals Price Indices Index
Composite Commodity Indices Fuel (Energy - oil, ng, coal, propane) Fuel (Energy) Index, 2016 = 100, includes Crude oil (petroleum), Natural Gas, Coal Price and Propane Indices Index
Gas Nat. Gas (Euro, Jap, American) Natural Gas Price Index, 2016 = 100, includes European, Japanese, and American Natural Gas Price Indices Index
Gas Nat. Gas (Indonesian LNG in Japan) Natural Gas, Indonesian Liquefied Natural Gas in Japan, US$ per Million Metric British Thermal Unit USD
Composite Commodity Indices Fuel & Non-Fuel Commodities All Commodity Price Index, 2016 = 100, includes both Fuel and Non-Fuel Price Indices Index
Nrg. Transition Input Material Prices Nrg. Transition Metals Composite Energy Transition Metal Index Index
Nrg. Transition Input Material Prices Battery-Grade Lithium Lithium Metal =99%, Battery Grade USD
Nrg. Transition Input Material Prices Manganese Manganese Electro CIF NWE (US$ perMT) USD
Nrg. Transition Input Material Prices Rare Earth Carbonate Rare Earth Carbonate REO 42-45 Dom. USD
Nrg. Transition Input Material Prices Silicon Lumps Silicon Lumps CIF NWE (US$ perMT) USD
Nrg. Transition Input Material Prices Vanad. Pentox. Vanadium Pentoxide CIF NWE USD

Net Imports

Imports and export flows of electricity from ENTSOE Transparency Platform.
Physical flows between bidding zones per market time unit.
Physical flow is defined as the measured real flow of energy between neighbouring bidding zones on the cross borders.

Note 1: Net imports still has some missing data and I’m not quite sure where it comes from (see Austria, Denmark, France, Switzerland).
Note 2: Also, the seeming upper and lower limits must have to do with capacity constraints on imports and exports (see Belgium, Norway, Sweden).

Day-ahead Solar and Wind Forecasts

A forecast of wind and solar power generation (MW) per country per hour of the following day data from ENTSOE Transparency Platform.

Note: we need to consider the logical link between these forecasts and price-setting/auction behaviour.

Redispatching Costs

Four transmission system operators (TSOs) in Germany actively monitor the contracted transmission of electricity in the grid to ensure load-balancing “in the event of imminent overloads in the electricity grid.”

“Redispatch is a request to adjust the active power feed-in of plants (conventional power plants, CHP plants, renewable energy plants, storage facilities) by the grid operator with the aim of avoiding or eliminating any bottlenecks that occur. This measure can be applied within and across control areas. By reducing the active power feed-in of one or more systems while simultaneously increasing the active power feed-in of one or more other systems, the total active power feed-in remains almost unchanged while at the same time relieving a bottleneck” (NETZTRANSPARENZ.DE).

(NETZTRANSPARENZ.DE) provide data on redispatch measures for the average, maximum, and per-quarter-hour(??) MW reduction. I think the latter is most relevant. See example at the link above.

The plots below show the associated “redispatching work” by primary energy type (conventional, renewable, miscellaneous, and unspecified).

Note: This information is also available from ENTSO-E. We might consider using this data source instead for consistency.
Note: to summarise by time period, I use either the mean or sum total of the measure of “redispatching work in mwh.” I am still not 100% sure what this means and whether this measure is most appropriate (or best summarised per hour as a sum or mean). We MUST revisit this data before implementation - NON-NEGOTIABLE.

Combining Data

For transparency’s sake, the code is below:
1. merge price, generation data by date-time and country-code
2. upper limit of TS to end of 2024
3. log transform with x + 1 to avoid zero value issues
4. create “share” values by dividing any generation variable by total load
5. for monthly commodity price values I fill these variables forward
6. Create hour and month dummies

Note: Potential transformation issues to discuss:
1. Forward-fill of monthly commodity price values
2. To deal with 0 values for certain variables I need to log_transform the value + 1. I don’t think this is a major issue given the magnitude of these variables but worth reconsidering.
3. (!!) For net imports there are several observations for country-specific flows that are NA. Most result from NA values reported for both imports and exports at those hours. However, for Denmark (1,860 obs) and France (2,234 obs) this issue results from non-NA imports and NA exports. THIS WILL ABSOLUTELY NEED TO BE REINVESTIGATED - WE CANNOT FORMALISE ANY RESULTS WITHOUT FINDING A WAY TO DEAL WITH THIS.

de_all <- de_hourly_prices %>% 
  rename(date_utc = datetime_utc) %>% 
  left_join(., entsoe_de, by = c("date_utc", "country")) %>% 
  left_join(., de_generation, by = c("date_utc", "country", "country_code")) %>% 
  arrange(date_utc) %>% 
  select(-starts_with("sum_")) %>% 
  filter(date_utc < "2024-03-31 02:00:00" & !is.na(gen_biomass_mw)) %>% 
  mutate(across(contains("gen"), ~log(. + 1), .names = "log_{.col}"),
         across(starts_with("gen"), ~./load, .names = "shr_{.col}"),
         log_load = log(load+1)) %>% 
  left_join(., imp_exp, by = "date_utc") %>% 
  mutate(across(contains("net_imports"), ~ifelse(is.na(.), 0, .)),
         shr_net_imports = total_net_imports/load) %>% 
  left_join(., fcast_sw, by = "date_utc") %>% 
  left_join(., redisp_df, by = "date_utc")

de_full <- de_all %>% 
  left_join(., ng_price, by = "date_utc") %>% 
  # We fill the monthly values down! 
  fill(PNGASEU, .direction = "down") %>% 
  fill(PALLFNF, .direction = "down") %>% 
  fill(PINDU, .direction = "down") %>% 
  # Given our redispatching data covers the 2015-2024 time period, it is safe to assume that NA values are 0 values. Barring the data/metric choice issues outlined in the "Redispatching Costs" tab above, of course.
  mutate(redisp_work_mwh = ifelse(is.na(redisp_work_mwh), 0, redisp_work_mwh),
         # Create hourly and monthly dummies
        hour = as.factor(hour(datetime_local)),
         month = as.factor(month(datetime_local)),
        log_redisp_mwh = log(redisp_work_mwh + 1),
        log_PNGASEU = log(PNGASEU + 1),
        log_PALLFNF = log(PALLFNF + 1),
        log_PINDU = log(PINDU + 1)) %>% 
  dummy_cols(select_columns = c("hour", "month"), remove_first_dummy = TRUE, remove_selected_columns = TRUE)

(TBA) Auction Bid Data

(TBA) Economic Activity

Approaching a model…

Below, I incorporate most explanatory variables (except solar and wind forecasts) above into models of the mean (not yet variance). The first model tab incorporates the variables simply as (log) levels whereas the second incorporates them as shares of total “load.” In each tab, I run the model with the same set of regressors to be selected over but with or without a log.ewma argument. In line with Rintamaki et al, we incorporate moving average terms for half-day, daily, and weekly price volatility (levels?) via log.ewma argument (correct?). I need to think more closely about how that functions together with the hourly and monthly dummies.

Note: log.ewma() for half-day, daily, and weekly price volatilities? Correct idea?
Note: Hourly and daily dummies the best way to incorporate “seasonality”? How might this interfere with the ewma terms?
Note: I do not currently incorporate any ARCH terms.
Note: day-ahead versus current prices in all variables? - are we not interested in deviations from forecasts for price-setting?
Note: Finally, a basic question, but I do not currently have the output variable (hourly wholesale electricity prices) in logs as it frequently takes negative values)

# Check to make sure data is ordered
de_full %>% arrange(date_utc) %>% all.equal(de_full)
## [1] TRUE

Levels as Exp. Vars.

reg_mat <- as.matrix(select(de_full,
                                       'log_load', 
                                       #'log_gen_biomass_mw',
                                       'log_gen_gas_mw', 
                                       'log_gen_coal_mw', 
                                       'log_gen_hydro_mw',
                                       'log_gen_geoth_mw', 
                                       'log_gen_nuclear_mw', 
                                       'log_gen_oth_ren_mw', 
                                       'log_gen_solar_mw', 
                                       'log_gen_wind_mw',
                                       'log_PNGASEU', 
                                       'log_PALLFNF',
                                       'log_PINDU',
                                       contains('hour'), 
                                       contains('month'),
                                       "net_imports_SE",
                                       "net_imports_PL",
                                       "net_imports_CZ",
                                       "net_imports_BE",
                                       "net_imports_CH",
                                       "net_imports_NO",
                                       "net_imports_NL",
                                       "net_imports_LU",
                                       "net_imports_FR",
                                       "net_imports_DK",
                                       "net_imports_AT",
                                       "log_redisp_mwh"
                                       #"total_net_imports" # Excluded as its sum components are each of the net_imports above
                                       )) # 'gen_wind_mw' 'gen_coal_mw', 'gen_hydro_mw',

de_mod_levs <- arx(de_full$price_eur_m_whe, 
              # Question: In theory, the ARCH(24) would proxy the seasonal (daily) AR term?
              #arch = 1:24, 
              mxreg = reg_mat)

de_mod_levs_logewma <- arx(de_full$price_eur_m_whe, 
              # As in Rintamaki et al. the below line incorporates moving average terms for the half-day (12), day (24 hours) and week (168 hours)  - correct?
              log.ewma = c(12, 24, 168),
              # Question: In theory, the ARCH(24) would proxy the seasonal (daily) AR term?
              #arch = 1:24, 
              mxreg = reg_mat)

# model selection and plotting of residuals
de_levs_sel <- getsm(de_mod_levs, ar.LjungB = NULL, arch.LjungB = NULL)
## 
## GUM mean equation:
## 
##                    reg.no. keep        coef   std.error    t-stat   p-value    
## mconst                   1    0  3.2813e+02  2.0264e+01   16.1926 < 2.2e-16 ***
## log_load                 2    0  1.4264e+01  1.7982e+00    7.9327 2.174e-15 ***
## log_gen_gas_mw           3    0  2.9921e+01  4.1248e-01   72.5390 < 2.2e-16 ***
## log_gen_coal_mw          4    0 -1.0527e+01  6.9903e-01  -15.0591 < 2.2e-16 ***
## log_gen_hydro_mw         5    0 -4.2343e+00  4.9264e-01   -8.5953 < 2.2e-16 ***
## log_gen_geoth_mw         6    0  1.4893e+01  7.3486e-01   20.2670 < 2.2e-16 ***
## log_gen_nuclear_mw       7    0 -2.3938e+01  6.8937e-01  -34.7246 < 2.2e-16 ***
## log_gen_oth_ren_mw       8    0  1.4739e+01  7.9203e-01   18.6093 < 2.2e-16 ***
## log_gen_solar_mw         9    0  4.9121e-01  8.9646e-02    5.4795 4.280e-08 ***
## log_gen_wind_mw         10    0  6.2983e-01  2.5612e-01    2.4591 0.0139312 *  
## log_PNGASEU             11    0  8.5731e+01  9.4949e-01   90.2918 < 2.2e-16 ***
## log_PALLFNF             12    0  9.4051e+01  3.5942e+00   26.1672 < 2.2e-16 ***
## log_PINDU               13    0 -2.2499e+02  2.1116e+00 -106.5507 < 2.2e-16 ***
## hour_1                  14    0 -1.7632e+00  1.0281e+00   -1.7149 0.0863622 .  
## hour_2                  15    0 -5.8969e-01  1.0359e+00   -0.5693 0.5691754    
## hour_3                  16    0 -9.6671e-01  1.0444e+00   -0.9256 0.3546674    
## hour_4                  17    0  3.1438e-01  1.0517e+00    0.2989 0.7649997    
## hour_5                  18    0  3.1168e+00  1.0512e+00    2.9651 0.0030273 ** 
## hour_6                  19    0  9.6354e+00  1.0577e+00    9.1100 < 2.2e-16 ***
## hour_7                  20    0  1.4310e+01  1.0842e+00   13.1983 < 2.2e-16 ***
## hour_8                  21    0  1.1163e+01  1.1569e+00    9.6494 < 2.2e-16 ***
## hour_9                  22    0  4.8684e+00  1.2666e+00    3.8435 0.0001214 ***
## hour_10                 23    0  9.3291e-02  1.3436e+00    0.0694 0.9446430    
## hour_11                 24    0 -2.4434e+00  1.3952e+00   -1.7513 0.0798985 .  
## hour_12                 25    0 -6.0279e+00  1.4129e+00   -4.2664 1.989e-05 ***
## hour_13                 26    0 -8.7513e+00  1.4188e+00   -6.1680 6.952e-10 ***
## hour_14                 27    0 -6.4458e+00  1.4069e+00   -4.5815 4.624e-06 ***
## hour_15                 28    0 -1.9415e+00  1.3882e+00   -1.3986 0.1619377    
## hour_16                 29    0  3.6883e+00  1.3592e+00    2.7136 0.0066581 ** 
## hour_17                 30    0  1.3920e+01  1.3180e+00   10.5614 < 2.2e-16 ***
## hour_18                 31    0  1.8904e+01  1.2515e+00   15.1049 < 2.2e-16 ***
## hour_19                 32    0  1.5360e+01  1.2098e+00   12.6967 < 2.2e-16 ***
## hour_20                 33    0  8.0379e+00  1.1584e+00    6.9391 3.979e-12 ***
## hour_21                 34    0  3.5557e+00  1.1126e+00    3.1957 0.0013954 ** 
## hour_22                 35    0  3.4225e+00  1.0698e+00    3.1992 0.0013785 ** 
## hour_23                 36    0 -2.3987e+00  1.0416e+00   -2.3029 0.0212866 *  
## month_2                 37    0  5.2213e+00  6.9419e-01    7.5214 5.480e-14 ***
## month_3                 38    0  1.2214e+01  6.9551e-01   17.5606 < 2.2e-16 ***
## month_4                 39    0  1.4991e+01  7.9216e-01   18.9247 < 2.2e-16 ***
## month_5                 40    0  1.6877e+01  8.9936e-01   18.7651 < 2.2e-16 ***
## month_6                 41    0  2.8414e+01  9.4585e-01   30.0410 < 2.2e-16 ***
## month_7                 42    0  2.8659e+01  9.1942e-01   31.1707 < 2.2e-16 ***
## month_8                 43    0  4.6855e+01  9.0613e-01   51.7093 < 2.2e-16 ***
## month_9                 44    0  3.0546e+01  8.2174e-01   37.1717 < 2.2e-16 ***
## month_10                45    0  6.8512e+00  7.6636e-01    8.9399 < 2.2e-16 ***
## month_11                46    0  1.2208e-01  7.2122e-01    0.1693 0.8655874    
## month_12                47    0  1.3598e+01  7.1350e-01   19.0575 < 2.2e-16 ***
## net_imports_SE          48    0  3.6400e-03  5.7645e-04    6.3145 2.725e-10 ***
## net_imports_PL          49    0 -8.7696e-03  4.1434e-04  -21.1650 < 2.2e-16 ***
## net_imports_CZ          50    0  7.6347e-03  2.4742e-04   30.8570 < 2.2e-16 ***
## net_imports_BE          51    0  2.0309e-02  4.7000e-04   43.2104 < 2.2e-16 ***
## net_imports_CH          52    0  4.8365e-03  1.6588e-04   29.1573 < 2.2e-16 ***
## net_imports_NO          53    0  4.3056e-02  4.1138e-04  104.6608 < 2.2e-16 ***
## net_imports_NL          54    0  6.7013e-03  1.3600e-04   49.2753 < 2.2e-16 ***
## net_imports_LU          55    0 -8.9564e-03  8.8479e-04  -10.1226 < 2.2e-16 ***
## net_imports_FR          56    0 -6.0007e-03  1.2437e-04  -48.2486 < 2.2e-16 ***
## net_imports_DK          57    0  1.9052e-03  1.7998e-04   10.5857 < 2.2e-16 ***
## net_imports_AT          58    0 -5.1762e-05  2.7211e-04   -0.1902 0.8491369    
## log_redisp_mwh          59    0 -4.7652e-01  5.1344e-02   -9.2809 < 2.2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## 
## Diagnostics:
## 
##                   Chi-sq df p-value
## Ljung-Box AR(1)    66776  1       0
## Ljung-Box ARCH(1)  63312  1       0
## 9 path(s) to search
## Searching: 1 2 3 4 5 6 7 8 9
## 
##   Path 1: 14 16 58 46 15 23 17 28 24 
##   Path 2: 15 46 58 23 17 16 14 28 24 
##   Path 3: 16 15 58 46 23 17 14 28 24 
##   Path 4: 17 23 46 58 15 16 14 28 24 
##   Path 5: 23 46 58 17 15 16 14 28 24 
##   Path 6: 24 15 58 46 28 16 14 36 17 
##   Path 7: 28 15 58 46 16 24 14 36 17 
##   Path 8: 46 23 58 17 15 16 14 28 24 
##   Path 9: 58 23 46 17 15 16 14 28 24 
## 
## Terminal models:
## 
##                 info(sc)      logl     n  k
## spec 1 (1-cut): 10.14578 -368209.8 72639 50
## spec 2:         10.14576 -368209.0 72639 50
## 
## Retained regressors (final model):
## 
##   mconst   log_load   log_gen_gas_mw   log_gen_coal_mw   log_gen_hydro_mw   log_gen_geoth_mw   log_gen_nuclear_mw   log_gen_oth_ren_mw   log_gen_solar_mw   log_gen_wind_mw   log_PNGASEU   log_PALLFNF   log_PINDU   hour_5   hour_6   hour_7   hour_8   hour_9   hour_10   hour_12   hour_13   hour_14   hour_16   hour_17   hour_18   hour_19   hour_20   hour_21   hour_22   month_2   month_3   month_4   month_5   month_6   month_7   month_8   month_9   month_10   month_12   net_imports_SE   net_imports_PL   net_imports_CZ   net_imports_BE   net_imports_CH   net_imports_NO   net_imports_NL   net_imports_LU   net_imports_FR   net_imports_DK   log_redisp_mwh
de_levs_sel %>% plot

de_levs_sel_logewma <- getsm(de_mod_levs_logewma, ar.LjungB = NULL, arch.LjungB = NULL)
## 
## GUM mean equation:
## 
##                    reg.no. keep        coef   std.error    t-stat   p-value    
## mconst                   1    0  3.2813e+02  2.0264e+01   16.1926 < 2.2e-16 ***
## log_load                 2    0  1.4264e+01  1.7982e+00    7.9327 2.174e-15 ***
## log_gen_gas_mw           3    0  2.9921e+01  4.1248e-01   72.5390 < 2.2e-16 ***
## log_gen_coal_mw          4    0 -1.0527e+01  6.9903e-01  -15.0591 < 2.2e-16 ***
## log_gen_hydro_mw         5    0 -4.2343e+00  4.9264e-01   -8.5953 < 2.2e-16 ***
## log_gen_geoth_mw         6    0  1.4893e+01  7.3486e-01   20.2670 < 2.2e-16 ***
## log_gen_nuclear_mw       7    0 -2.3938e+01  6.8937e-01  -34.7246 < 2.2e-16 ***
## log_gen_oth_ren_mw       8    0  1.4739e+01  7.9203e-01   18.6093 < 2.2e-16 ***
## log_gen_solar_mw         9    0  4.9121e-01  8.9646e-02    5.4795 4.280e-08 ***
## log_gen_wind_mw         10    0  6.2983e-01  2.5612e-01    2.4591 0.0139312 *  
## log_PNGASEU             11    0  8.5731e+01  9.4949e-01   90.2918 < 2.2e-16 ***
## log_PALLFNF             12    0  9.4051e+01  3.5942e+00   26.1672 < 2.2e-16 ***
## log_PINDU               13    0 -2.2499e+02  2.1116e+00 -106.5507 < 2.2e-16 ***
## hour_1                  14    0 -1.7632e+00  1.0281e+00   -1.7149 0.0863622 .  
## hour_2                  15    0 -5.8969e-01  1.0359e+00   -0.5693 0.5691754    
## hour_3                  16    0 -9.6671e-01  1.0444e+00   -0.9256 0.3546674    
## hour_4                  17    0  3.1438e-01  1.0517e+00    0.2989 0.7649997    
## hour_5                  18    0  3.1168e+00  1.0512e+00    2.9651 0.0030273 ** 
## hour_6                  19    0  9.6354e+00  1.0577e+00    9.1100 < 2.2e-16 ***
## hour_7                  20    0  1.4310e+01  1.0842e+00   13.1983 < 2.2e-16 ***
## hour_8                  21    0  1.1163e+01  1.1569e+00    9.6494 < 2.2e-16 ***
## hour_9                  22    0  4.8684e+00  1.2666e+00    3.8435 0.0001214 ***
## hour_10                 23    0  9.3291e-02  1.3436e+00    0.0694 0.9446430    
## hour_11                 24    0 -2.4434e+00  1.3952e+00   -1.7513 0.0798985 .  
## hour_12                 25    0 -6.0279e+00  1.4129e+00   -4.2664 1.989e-05 ***
## hour_13                 26    0 -8.7513e+00  1.4188e+00   -6.1680 6.952e-10 ***
## hour_14                 27    0 -6.4458e+00  1.4069e+00   -4.5815 4.624e-06 ***
## hour_15                 28    0 -1.9415e+00  1.3882e+00   -1.3986 0.1619377    
## hour_16                 29    0  3.6883e+00  1.3592e+00    2.7136 0.0066581 ** 
## hour_17                 30    0  1.3920e+01  1.3180e+00   10.5614 < 2.2e-16 ***
## hour_18                 31    0  1.8904e+01  1.2515e+00   15.1049 < 2.2e-16 ***
## hour_19                 32    0  1.5360e+01  1.2098e+00   12.6967 < 2.2e-16 ***
## hour_20                 33    0  8.0379e+00  1.1584e+00    6.9391 3.979e-12 ***
## hour_21                 34    0  3.5557e+00  1.1126e+00    3.1957 0.0013954 ** 
## hour_22                 35    0  3.4225e+00  1.0698e+00    3.1992 0.0013785 ** 
## hour_23                 36    0 -2.3987e+00  1.0416e+00   -2.3029 0.0212866 *  
## month_2                 37    0  5.2213e+00  6.9419e-01    7.5214 5.480e-14 ***
## month_3                 38    0  1.2214e+01  6.9551e-01   17.5606 < 2.2e-16 ***
## month_4                 39    0  1.4991e+01  7.9216e-01   18.9247 < 2.2e-16 ***
## month_5                 40    0  1.6877e+01  8.9936e-01   18.7651 < 2.2e-16 ***
## month_6                 41    0  2.8414e+01  9.4585e-01   30.0410 < 2.2e-16 ***
## month_7                 42    0  2.8659e+01  9.1942e-01   31.1707 < 2.2e-16 ***
## month_8                 43    0  4.6855e+01  9.0613e-01   51.7093 < 2.2e-16 ***
## month_9                 44    0  3.0546e+01  8.2174e-01   37.1717 < 2.2e-16 ***
## month_10                45    0  6.8512e+00  7.6636e-01    8.9399 < 2.2e-16 ***
## month_11                46    0  1.2208e-01  7.2122e-01    0.1693 0.8655874    
## month_12                47    0  1.3598e+01  7.1350e-01   19.0575 < 2.2e-16 ***
## net_imports_SE          48    0  3.6400e-03  5.7645e-04    6.3145 2.725e-10 ***
## net_imports_PL          49    0 -8.7696e-03  4.1434e-04  -21.1650 < 2.2e-16 ***
## net_imports_CZ          50    0  7.6347e-03  2.4742e-04   30.8570 < 2.2e-16 ***
## net_imports_BE          51    0  2.0309e-02  4.7000e-04   43.2104 < 2.2e-16 ***
## net_imports_CH          52    0  4.8365e-03  1.6588e-04   29.1573 < 2.2e-16 ***
## net_imports_NO          53    0  4.3056e-02  4.1138e-04  104.6608 < 2.2e-16 ***
## net_imports_NL          54    0  6.7013e-03  1.3600e-04   49.2753 < 2.2e-16 ***
## net_imports_LU          55    0 -8.9564e-03  8.8479e-04  -10.1226 < 2.2e-16 ***
## net_imports_FR          56    0 -6.0007e-03  1.2437e-04  -48.2486 < 2.2e-16 ***
## net_imports_DK          57    0  1.9052e-03  1.7998e-04   10.5857 < 2.2e-16 ***
## net_imports_AT          58    0 -5.1762e-05  2.7211e-04   -0.1902 0.8491369    
## log_redisp_mwh          59    0 -4.7652e-01  5.1344e-02   -9.2809 < 2.2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## GUM log-variance equation:
## 
##                    coef std.error  t-stat   p-value    
## vconst        -0.076130  0.038342  3.9425   0.04708 *  
## logEqWMA(12)   0.799385  0.013005 61.4696 < 2.2e-16 ***
## logEqWMA(24)   0.183992  0.015948 11.5372 < 2.2e-16 ***
## logEqWMA(168)  0.061593  0.010397  5.9243  3.15e-09 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Diagnostics:
## 
##                   Chi-sq df p-value
## Ljung-Box AR(1)    57845  1       0
## Ljung-Box ARCH(1)  25150  1       0
## 9 path(s) to search
## Searching: 1 2 3 4 5 6 7 8 9
## 
##   Path 1: 14 16 58 46 15 23 17 28 24 
##   Path 2: 15 46 58 23 17 16 14 28 24 
##   Path 3: 16 15 58 46 23 17 14 28 24 
##   Path 4: 17 23 46 58 15 16 14 28 24 
##   Path 5: 23 46 58 17 15 16 14 28 24 
##   Path 6: 24 15 58 46 28 16 14 36 17 
##   Path 7: 28 15 58 46 16 24 14 36 17 
##   Path 8: 46 23 58 17 15 16 14 28 24 
##   Path 9: 58 23 46 17 15 16 14 28 24 
## 
## Terminal models:
## 
##                 info(sc)      logl     n  k
## spec 1 (1-cut): 9.010506 -326977.2 72639 50
## spec 2:         9.010651 -326982.5 72639 50
## 
## Retained regressors (final model):
## 
##   mconst   log_load   log_gen_gas_mw   log_gen_coal_mw   log_gen_hydro_mw   log_gen_geoth_mw   log_gen_nuclear_mw   log_gen_oth_ren_mw   log_gen_solar_mw   log_gen_wind_mw   log_PNGASEU   log_PALLFNF   log_PINDU   hour_5   hour_6   hour_7   hour_8   hour_9   hour_12   hour_13   hour_14   hour_16   hour_17   hour_18   hour_19   hour_20   hour_21   hour_22   hour_23   month_2   month_3   month_4   month_5   month_6   month_7   month_8   month_9   month_10   month_12   net_imports_SE   net_imports_PL   net_imports_CZ   net_imports_BE   net_imports_CH   net_imports_NO   net_imports_NL   net_imports_LU   net_imports_FR   net_imports_DK   log_redisp_mwh
de_levs_sel_logewma %>% plot

Shares as Exp. Vars.

reg_mat_shares <- as.matrix(select(de_full, 
                                       #'shr_gen_biomass_mw',  
                                       'shr_gen_gas_mw', 
                                       'shr_gen_coal_mw',
                                       'shr_gen_hydro_mw',
                                       'shr_gen_geoth_mw', 
                                       'shr_gen_nuclear_mw', 
                                       'shr_gen_oth_ren_mw',
                                       'shr_gen_solar_mw',
                                       'shr_gen_wind_mw',
                                       'log_PNGASEU',
                                       'log_PALLFNF',
                                       'log_PINDU',
                                       contains('hour'), 
                                       contains('month'),
                                       "log_redisp_mwh",
                                       shr_net_imports)) # 'gen_wind_mw' 'gen_coal_mw', 'gen_hydro_mw',

de_mod_shares <- arx(de_full$price_eur_m_whe, 
              mxreg = reg_mat_shares) 

de_mod_shares_logewma <- arx(de_full$price_eur_m_whe, 
                     log.ewma = c(12, 24, 168),
                     mxreg = reg_mat_shares) 

de_shares_sel <- getsm(de_mod_shares, ar.LjungB = NULL, arch.LjungB = NULL)
## 
## GUM mean equation:
## 
##                    reg.no. keep        coef   std.error    t-stat   p-value    
## mconst                   1    0  342.157005   11.476998   29.8124 < 2.2e-16 ***
## shr_gen_gas_mw           2    0  506.020085    5.163188   98.0053 < 2.2e-16 ***
## shr_gen_coal_mw          3    0 -159.292461    4.174788  -38.1558 < 2.2e-16 ***
## shr_gen_hydro_mw         4    0  -69.768475    6.881020  -10.1393 < 2.2e-16 ***
## shr_gen_geoth_mw         5    0 4240.473991 2139.530578    1.9820 0.0474870 *  
## shr_gen_nuclear_mw       6    0 -363.784607    6.868355  -52.9653 < 2.2e-16 ***
## shr_gen_oth_ren_mw       7    0 9602.354101  338.542280   28.3638 < 2.2e-16 ***
## shr_gen_solar_mw         8    0  -71.195809    3.026692  -23.5226 < 2.2e-16 ***
## shr_gen_wind_mw          9    0 -130.492239    3.078198  -42.3924 < 2.2e-16 ***
## log_PNGASEU             10    0   87.007975    0.969775   89.7198 < 2.2e-16 ***
## log_PALLFNF             11    0  154.366176    3.650778   42.2831 < 2.2e-16 ***
## log_PINDU               12    0 -236.042424    2.278151 -103.6114 < 2.2e-16 ***
## hour_1                  13    0   -1.755027    1.138289   -1.5418 0.1231238    
## hour_2                  14    0   -1.497033    1.145231   -1.3072 0.1911528    
## hour_3                  15    0   -3.236108    1.153007   -2.8067 0.0050070 ** 
## hour_4                  16    0   -4.542115    1.158567   -3.9205 8.846e-05 ***
## hour_5                  17    0   -4.810603    1.160486   -4.1453 3.397e-05 ***
## hour_6                  18    0   -1.360620    1.195198   -1.1384 0.2549547    
## hour_7                  19    0    3.181494    1.231972    2.5824 0.0098124 ** 
## hour_8                  20    0    1.532457    1.254420    1.2216 0.2218453    
## hour_9                  21    0   -3.962872    1.264279   -3.1345 0.0017222 ** 
## hour_10                 22    0   -8.565204    1.290766   -6.6358 3.251e-11 ***
## hour_11                 23    0  -10.046090    1.328995   -7.5592 4.104e-14 ***
## hour_12                 24    0  -10.775257    1.346586   -8.0019 1.243e-15 ***
## hour_13                 25    0  -10.671208    1.360772   -7.8420 4.493e-15 ***
## hour_14                 26    0   -8.771060    1.356596   -6.4655 1.016e-10 ***
## hour_15                 27    0   -4.961998    1.338842   -3.7062 0.0002106 ***
## hour_16                 28    0   -0.812991    1.311922   -0.6197 0.5354611    
## hour_17                 29    0    7.161501    1.294859    5.5307 3.200e-08 ***
## hour_18                 30    0   11.833023    1.269717    9.3194 < 2.2e-16 ***
## hour_19                 31    0   12.164281    1.244801    9.7721 < 2.2e-16 ***
## hour_20                 32    0    6.631773    1.202592    5.5146 3.508e-08 ***
## hour_21                 33    0    0.017882    1.180560    0.0151 0.9879150    
## hour_22                 34    0   -1.739297    1.165327   -1.4925 0.1355619    
## hour_23                 35    0   -5.182766    1.150155   -4.5061 6.612e-06 ***
## month_2                 36    0    4.873959    0.762041    6.3959 1.605e-10 ***
## month_3                 37    0   13.940209    0.762768   18.2758 < 2.2e-16 ***
## month_4                 38    0   14.159736    0.836582   16.9257 < 2.2e-16 ***
## month_5                 39    0   14.477640    0.928678   15.5895 < 2.2e-16 ***
## month_6                 40    0   22.608480    0.946742   23.8803 < 2.2e-16 ***
## month_7                 41    0   24.020452    0.900792   26.6659 < 2.2e-16 ***
## month_8                 42    0   44.952552    0.892767   50.3520 < 2.2e-16 ***
## month_9                 43    0   31.260794    0.838983   37.2603 < 2.2e-16 ***
## month_10                44    0   13.069373    0.811585   16.1035 < 2.2e-16 ***
## month_11                45    0    7.290577    0.782804    9.3134 < 2.2e-16 ***
## month_12                46    0   15.754549    0.775938   20.3039 < 2.2e-16 ***
## log_redisp_mwh          47    0   -0.496050    0.056503   -8.7792 < 2.2e-16 ***
## shr_net_imports         48    0   14.310542    3.970849    3.6039 0.0003137 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## 
## Diagnostics:
## 
##                   Chi-sq df p-value
## Ljung-Box AR(1)    67854  1       0
## Ljung-Box ARCH(1)  63977  1       0
## 7 path(s) to search
## Searching: 1 2 3 4 5 6 7
## 
##   Path 1: 13 28 18 14 34 33 5 
##   Path 2: 14 28 18 34 13 33 5 
##   Path 3: 18 28 33 14 13 34 5 
##   Path 4: 20 33 28 14 18 13 34 -5 
##   Path 5: 28 33 18 14 13 34 5 
##   Path 6: 33 28 18 14 13 34 5 
##   Path 7: 34 28 18 14 13 33 5 
## 
## Terminal models:
## 
##                 info(sc)      logl     n  k
## spec 1 (1-cut): 10.35056 -375697.7 72639 41
## spec 2:         10.35050 -375695.5 72639 41
## 
## Retained regressors (final model):
## 
##   mconst   shr_gen_gas_mw   shr_gen_coal_mw   shr_gen_hydro_mw   shr_gen_nuclear_mw   shr_gen_oth_ren_mw   shr_gen_solar_mw   shr_gen_wind_mw   log_PNGASEU   log_PALLFNF   log_PINDU   hour_3   hour_4   hour_5   hour_7   hour_8   hour_9   hour_10   hour_11   hour_12   hour_13   hour_14   hour_15   hour_17   hour_18   hour_19   hour_20   hour_23   month_2   month_3   month_4   month_5   month_6   month_7   month_8   month_9   month_10   month_11   month_12   log_redisp_mwh   shr_net_imports
de_shares_sel %>% plot

de_shares_sel_logewma <- getsm(de_mod_shares_logewma, ar.LjungB = NULL, arch.LjungB = NULL)
## 
## GUM mean equation:
## 
##                    reg.no. keep        coef   std.error    t-stat   p-value    
## mconst                   1    0  342.157005   11.476998   29.8124 < 2.2e-16 ***
## shr_gen_gas_mw           2    0  506.020085    5.163188   98.0053 < 2.2e-16 ***
## shr_gen_coal_mw          3    0 -159.292461    4.174788  -38.1558 < 2.2e-16 ***
## shr_gen_hydro_mw         4    0  -69.768475    6.881020  -10.1393 < 2.2e-16 ***
## shr_gen_geoth_mw         5    0 4240.473991 2139.530578    1.9820 0.0474870 *  
## shr_gen_nuclear_mw       6    0 -363.784607    6.868355  -52.9653 < 2.2e-16 ***
## shr_gen_oth_ren_mw       7    0 9602.354101  338.542280   28.3638 < 2.2e-16 ***
## shr_gen_solar_mw         8    0  -71.195809    3.026692  -23.5226 < 2.2e-16 ***
## shr_gen_wind_mw          9    0 -130.492239    3.078198  -42.3924 < 2.2e-16 ***
## log_PNGASEU             10    0   87.007975    0.969775   89.7198 < 2.2e-16 ***
## log_PALLFNF             11    0  154.366176    3.650778   42.2831 < 2.2e-16 ***
## log_PINDU               12    0 -236.042424    2.278151 -103.6114 < 2.2e-16 ***
## hour_1                  13    0   -1.755027    1.138289   -1.5418 0.1231238    
## hour_2                  14    0   -1.497033    1.145231   -1.3072 0.1911528    
## hour_3                  15    0   -3.236108    1.153007   -2.8067 0.0050070 ** 
## hour_4                  16    0   -4.542115    1.158567   -3.9205 8.846e-05 ***
## hour_5                  17    0   -4.810603    1.160486   -4.1453 3.397e-05 ***
## hour_6                  18    0   -1.360620    1.195198   -1.1384 0.2549547    
## hour_7                  19    0    3.181494    1.231972    2.5824 0.0098124 ** 
## hour_8                  20    0    1.532457    1.254420    1.2216 0.2218453    
## hour_9                  21    0   -3.962872    1.264279   -3.1345 0.0017222 ** 
## hour_10                 22    0   -8.565204    1.290766   -6.6358 3.251e-11 ***
## hour_11                 23    0  -10.046090    1.328995   -7.5592 4.104e-14 ***
## hour_12                 24    0  -10.775257    1.346586   -8.0019 1.243e-15 ***
## hour_13                 25    0  -10.671208    1.360772   -7.8420 4.493e-15 ***
## hour_14                 26    0   -8.771060    1.356596   -6.4655 1.016e-10 ***
## hour_15                 27    0   -4.961998    1.338842   -3.7062 0.0002106 ***
## hour_16                 28    0   -0.812991    1.311922   -0.6197 0.5354611    
## hour_17                 29    0    7.161501    1.294859    5.5307 3.200e-08 ***
## hour_18                 30    0   11.833023    1.269717    9.3194 < 2.2e-16 ***
## hour_19                 31    0   12.164281    1.244801    9.7721 < 2.2e-16 ***
## hour_20                 32    0    6.631773    1.202592    5.5146 3.508e-08 ***
## hour_21                 33    0    0.017882    1.180560    0.0151 0.9879150    
## hour_22                 34    0   -1.739297    1.165327   -1.4925 0.1355619    
## hour_23                 35    0   -5.182766    1.150155   -4.5061 6.612e-06 ***
## month_2                 36    0    4.873959    0.762041    6.3959 1.605e-10 ***
## month_3                 37    0   13.940209    0.762768   18.2758 < 2.2e-16 ***
## month_4                 38    0   14.159736    0.836582   16.9257 < 2.2e-16 ***
## month_5                 39    0   14.477640    0.928678   15.5895 < 2.2e-16 ***
## month_6                 40    0   22.608480    0.946742   23.8803 < 2.2e-16 ***
## month_7                 41    0   24.020452    0.900792   26.6659 < 2.2e-16 ***
## month_8                 42    0   44.952552    0.892767   50.3520 < 2.2e-16 ***
## month_9                 43    0   31.260794    0.838983   37.2603 < 2.2e-16 ***
## month_10                44    0   13.069373    0.811585   16.1035 < 2.2e-16 ***
## month_11                45    0    7.290577    0.782804    9.3134 < 2.2e-16 ***
## month_12                46    0   15.754549    0.775938   20.3039 < 2.2e-16 ***
## log_redisp_mwh          47    0   -0.496050    0.056503   -8.7792 < 2.2e-16 ***
## shr_net_imports         48    0   14.310542    3.970849    3.6039 0.0003137 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## GUM log-variance equation:
## 
##                    coef std.error  t-stat   p-value    
## vconst        -0.154083  0.040555 14.4350 0.0001451 ***
## logEqWMA(12)   0.719664  0.012424 57.9237 < 2.2e-16 ***
## logEqWMA(24)   0.293390  0.015561 18.8546 < 2.2e-16 ***
## logEqWMA(168)  0.041720  0.010367  4.0242 5.723e-05 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Diagnostics:
## 
##                   Chi-sq df p-value
## Ljung-Box AR(1)    61587  1       0
## Ljung-Box ARCH(1)  31776  1       0
## 7 path(s) to search
## Searching: 1 2 3 4 5 6 7
## 
##   Path 1: 13 28 18 14 34 33 5 
##   Path 2: 14 28 18 34 13 33 5 
##   Path 3: 18 28 33 14 13 34 5 
##   Path 4: 20 33 28 14 18 13 34 -5 
##   Path 5: 28 33 18 14 13 34 5 
##   Path 6: 33 28 18 14 13 34 5 
##   Path 7: 34 28 18 14 13 33 5 
## 
## Terminal models:
## 
##                 info(sc)      logl     n  k
## spec 1 (1-cut): 9.251739 -335789.1 72639 41
## spec 2:         9.251361 -335775.3 72639 41
## 
## Retained regressors (final model):
## 
##   mconst   shr_gen_gas_mw   shr_gen_coal_mw   shr_gen_hydro_mw   shr_gen_nuclear_mw   shr_gen_oth_ren_mw   shr_gen_solar_mw   shr_gen_wind_mw   log_PNGASEU   log_PALLFNF   log_PINDU   hour_3   hour_4   hour_5   hour_7   hour_8   hour_9   hour_10   hour_11   hour_12   hour_13   hour_14   hour_15   hour_17   hour_18   hour_19   hour_20   hour_23   month_2   month_3   month_4   month_5   month_6   month_7   month_8   month_9   month_10   month_11   month_12   log_redisp_mwh   shr_net_imports
de_shares_sel_logewma %>% plot

# # Perform gets model selection over the log-variance model above
# de_vmod <- getsv(de_mod, t.pval = 0.001, ar.LjungB = NULL)
# 
# de_vmod %>% plot
# 
# data_de %>% filter(de_price < 0)

(TBA) Spotlight on UK, Germany, Denmark

Here, see the time series for Germany and Denmark (to compare to the data avialable in Rintamaki et al. - color portion of the TS plots). UK spotlight to discuss potential interest to David.

(TBA) Full Extension to Europe

Note: Recall also that you can find our replication of Rintamaki et al. here. We have yet to analyse the differences between the hourly volatility specification using getsv() versus the Rintamaki results.

Hourly Prices

Using data from EMBER on European wholesale electricity prices (hourly - daily and monthly also exist), The hourly prices are the day-ahead wholesale prices by country.

The following is data from ENTSO-E on hourly electricity loads.

## [1] TRUE

Daily and Monthly Prices